[None][feat] GEN slot preemption: pause decode requests in place to admit at-risk waiters - #21334

… in place

GEN slot preemption pauses a decode request in place: it stays resident
but is left out of the scheduled batch, and its id reaches the spec
metadata as paused_in_place_ids. The DSv4 DSpark rolling-window path
already keeps such a request's window, but the standalone DSpark worker
(DSparkWorker) and DFlash run DFlashSpecMetadata.prepare, which frees the
drafter context slot of every request missing from the batch. A paused
request then lost its slot, context length and chunk position, and
drafted after resume from an empty context like a fresh disagg GEN
arrival.

- DFlashSpecMetadata.prepare keeps the slots of paused_in_place_ids; a
  request that finishes after a pause (neither batched nor paused) is
  still released.
- DFlashWorker sizes its request slots like the other per-resident pools:
  max_batch plus gen_preemption.resident_headroom() (0 when preemption
  is off, so nothing changes by default), with the dummy slot after
  them. The private ctx arena, the per-slot context lengths and the
  block-table rows follow num_slots; _batch_to_slot stays per batch row.

Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>

…MPT_LATE_TRIGGER)

GEN slot preemption pauses a running decode request only when the GEN
admission queue's next pop is in the slack tier with slack against the
adaptive target X in [0, TLLM_GEN_PREEMPT_SLACK_S]. A waiter that is
already late (slack < 0), or a next pop from the guard or breaker tier,
never triggers a pause, so under sustained overload the trigger goes
quiet exactly when the queue is furthest behind.

TLLM_GEN_PREEMPT_LATE_TRIGGER=1 drops the lower bound and the tier
condition: the next pop is at risk whenever it is keyed and its slack is
at most SLACK_S, late or not, and num_at_risk counts every keyed waiter
in that range. The trigger stays head-only and the per-pass budget is
unchanged. Default 0 keeps the current behaviour. The flag is a
GenPreemptionConfig field, so the startup "enabled" line shows it.

Meant to run with TLLM_GEN_PREEMPT_MAX_PER_REQUEST=2.

Tests: late and guard-tier next pops trigger only with the flag, an
on-time next pop with too much slack never does; a request is paused
twice through the executor pass (resume lane, hysteresis) and its third
pause is refused at max_per_request=2; the startup line shows the knobs.

Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>

…objective (laxity victims, sacrifice credit)

target_output_tps no longer converts to a TPOT budget (max(0, R0 / T - 1)).
It is now a service-level objective: T is a per-request output-rate floor
that at least 1 - eps of the requests meet (eps = 0.2, an internal constant).
tpot_budget keeps the phase-21 controller and wait victims unchanged.

Policy (k3-disagg-sim phase 24, SIMULATION 6.25):
- laxity L = t_first + (max_tokens - 1) / T - (now + remaining / r), r the
  request's own unpaused rate;
- victim: the largest L among requests with L >= 0.02 D and < 8 pauses,
  resumed when L is down to 0.01 D (or after 0.35 D); else a request
  already below T (L < 0 or sacrificed) with < 3 pauses, longest resident;
  else, with credit >= 1, sacrifice the longest resident (credit -= 1);
- credit per completion after X: clamp(credit + eps - [below T]
  + [sacrificed], +-3000 eps);
- resume lane: earliest forced-resume time first (least laxity);
- window fixed at 0.15 D (no controller), cap 0.375 x
  max_num_active_requests, late trigger on.
All inputs are rank-identical (token counts, max_tokens, completions, rank
0's broadcast clock); no new collective. The 30 s rank-0 stat line adds
below_target (window / total), credit, laxity / below-target / sacrificed
victims, mean laxity, laxity resumes and paused below-target requests.

Sim, 16 seeds vs off, T = control / 1.10: c220 DWDP16 p90 -2.69 s, TPOT
+5.7%, 12% of requests below T; DEP16 -2.02 s / +4.6% / 12%. c250 DWDP16
-3.51 s / +8.1%, DEP16 -3.35 s / +8.5%. Total tps within +-0.5%. E2E p90
0.3-1.9 s worse than tpot_budget at c220-c250. Not GPU-tested.

Tests: tpot_budget tests no longer go through the conversion; new
target_output_tps tests for laxity, victim classes and pause limits, the
laxity-margin forced resume, the credit bucket, least-laxity resume order,
no controller, the stat line, an executor pass and rank-identical inputs.

Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>

…m the change

Same user-facing config (tpot_budget / target_output_tps) and the same
decisions; the change is about half its previous size.

- gen_preemption.py: one per-request state object, one GenPreemption.step()
  entry point per scheduling pass (observe, resume, pause), the two modes as
  two victim strategies over shared pause/resume bookkeeping, a single
  resume lane ordered by due time, a short stat line, and executor setup
  checks (create_gen_preemption) moved out of PyExecutor.
- The late trigger is always on (it was never configurable), so
  GenAdmissionWaitingQueue.num_at_risk drops its flag.
- Pools add resident_headroom() unconditionally (it is 0 when preemption is
  off, and pipeline parallelism is rejected when it is on).
- Shorter llm_args descriptions and docs; focused, parametrized tests.

Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>

…e two user fields

GenPreemptionSettings now holds only tpot_budget / target_output_tps; the
policy constants are class attributes of GenPreemption and the paused cap is
a module function. No behavior change.

Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>

…ption-pr

Conflicts: PyExecutor.__init__ and SchedulerConfig fields (kept both sides);
test_gen_preemption.py follows main's tests/unittest/_torch/executor ->
tests/unittest/_torch/pyexecutor move.

Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>